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NASA IV&V Program Overview 


¢ Responsible for performing Independent Verification 
and Validation (IV&V) on high profile NASA missions 


¢ Functionally reports to the Chief, Office of Safety and 
Mission Assurance (OSMA) - NASA HQ 


¢ Agency requirements for IV&V defined in 


— NPD 7120.4D, NASA Engineering and Program/Project Management 
Policy 
— NPR7150.2, NASA Software Engineering Requirements 


¢ Staff of ~250 personnel 

¢ ISO 9001:2008 Certified 

¢ Only NASA Center with VPP Star Certification 
¢ Pursuing FedRamp Certification 


gy 


¢ Legacy of Customer Satisfaction 
— 94% acceptance rate for findings 
— 95% customer satisfaction (2015 annual customer survey) 
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NASA IV&V Process 


Reassessment 


| + IV&V Findings 
| * Dev Schedule Changes 


* More information 


“Eollow the Risk” 


¥ 
System Assurance : Assurance 

Understanding Assessment Design camel Conclusions 

* Heritage Review 

* Understand * Scoring Criteria * DetermineNecessaryRigor * MercyRuleAssessment * Document Assurance Gained 
* Mission Goals Generation/Refinement * Analysisstrategy to meet * Perform Analysis * ReportAnalysis Conclusions 
* Mission Concept * Cap Risk Ranking/Binning rigor needs * Capture Evidence (DMA) * Articulate Residual Risk 
* Developer * SyslevelAO’sforPriorityCaps *° StaffingPlan * Document concerns 
* Technology * AODecomp or Correlation * Schedule 
* Mission Capabilities * RankingofAO’s * Budget Considerations 

* Mission, System, SW 

Architecture 


Technical Reference Assurance Required Assurance Planned Technical Products Assurance Conclusions 


¢ Independent Testing — Application of dynamic 
analysis that is conducted by an organization apart 
from the system developer to verify the 
implementation (source/object code) is correct 
and complete 
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¢ Independent Software Testing is a method used 
by our Program to reduce system and software 
risks 


¢ Provides evidence-based assurance of correct and 
complete implementation of software behaviors in 
final source or binary images within an 
operational environment 


¢ Provides substantiating evidence of issues 
discovered during other activities in the IV&V 
process 
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Testing provides strong objective evidence of defects 


Supplements Requirements Analysis, Design Analysis, and 
Implementation Analysis 


Supplements Test Documentation Analysis for integration coverage 


Provides high degree of assurance that assertions regarding 
software quality and suitability are accurate 


Practical method for verifying certain behaviors, like: 


Deadlock and race avoidance 

Inter-process timing and queuing (like SpaceWire backpressure) 
Complex fault management with multiple faults 

Extended duration operations 

Hardware failure fault management 

Interdependencies 
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Plan the Testing 


— Establishing the focus areas for testing; what assurance objectives are best 
accomplished via testing; resources needed to support the analysis and 
testing 


— Determining the requirements of the test platform to support the planned 
assurance objectives 


Acquire Test Environment 

— Acquire the simulations, test hardware and other assets 

— Integrate all into functional test bed for injecting IV&V tests scripts 
Prepare for Testing 

— Configuration of the test environment 

— Develop test plans and associated test scripts 

— Configuration management of test environment and artifacts 
Perform Testing 

— Execute test, collect results and evaluate results 
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¢ Consider risk of system/subsystem 
— IV&V is Risk-Driven and Focused 
— IV&V analyses should “Follow the Risk” 


¢ Select appropriate IV&V artifacts and 
development artifacts as test source data 


¢ Identify a set of developer tests to validate 
test environment 


¢ Plan and document test cases 
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¢ Project 1 
— IV&V Analysis team comes up with candidate test cases in lightweight form 


— Candidate test cases are screened through Independent Test Working 
Group and approved by lead 


— Selected Test Cases are fully developed and tested 


¢ Project 2 


— Team identifies risk reduction scenarios (RRS) which are not being tested 
by the Developer 


— The RRSs are specified in the test plan 


¢ Project 3 
— Analysts define test cases based on lifecycle analysis being performed 
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¢ Definition of Risk Reduction Scenarios: 


— Risk scenarios must have impact to the mission success or cause the loss of mission 
objectives. For example, a fault that happened at the wrong time may cause the loss 


of mission. 
Draft / Perform Risk 
i Peer Review of Close Risk/Issue 
Propose Scenario Analysis onthe 9 =———> eee 
Reise | anereue scenario 
eyec 
Scenario Approval ch 

1 urtner 

Bove Actions 


Run Tests to verify 


aie Findings 
Draft Test 
Procedures & Scripts 
(Dry-Run Optional) Record Findings / 
| Reject Test Results Analysis __| Issues 
Test Procedure & 


Script Peer Review 


Pavorove 


Run Test 


Database 
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¢ Some Examples 


Stored Command Sequence (SCS) Validation 
Long-Duration FSW Testing 

Fault Scenarios (e.g., Stuck Thruster) 

Flight Computer fault injection (Instruction faults, etc.) 
Primary/Backup C&DH Swapping 

Dropped Packet Analysis 

Visualization Capability for Attitude Control System 


Risk 


* Prior IV&V/Project Risk captures 
complexity and criticality of 


Mitigation/Action =i 


olor’ comand Fea nces ¢ Mitigation included more rigorous Results/Findings 
when also considered a part of aa al 
the larger integrated system of Sil orihotson seine rns saee oer 
: 5 and “potential” for additional * Test Scenarios Tested 
PUeM au: testing — 15 Issues Ide ed to Date 


* IV&V perform additional SCS 
testing 


* Tie into developer Working Group 


Value Added 


* Impacts to Project 
* Impact to schedule and cost 
averted. 


* Impacts to IV&V 
* Insight into FM/Observatory 
response increased. 
¢ Awareness of SCS criticality 
revisited 
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Develop, maintain, and operate test environments and 
supporting tools for the IV&V Program that enables the dynamic 
analysis of software behaviors for multiple NASA missions 
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@ NASA Operational Simulator 


5a SiilM ee for Small Satellites 
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Test Environment Development Process 


Test System Assets 
(FSW Source, Tests, 
Specs, Emulators,...) 


Test System 
yee) e)iayed 


id F-Talallay-a-lare, 
Analysis 


WV i Te F-h a= 
Test System 


Closeout 
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Build, Test, 


Kickoff Meeting 
Test Readiness 
Review 

Release Readiness 
Review 

Customer Feedback 
/ Post-Mortem 
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Test Environment Types 


I) Software only II) HW/SW Hybrid III) Software Only - Alt IV) Subsystem - Alt 


c NTOUT Danaram 
SA IV&V Program 


Simulation - Full-Up Simulation - Full-Up Compile Execution Compilation 
Requires Hardest to build Not TEST as Fly ¢ Least test 
moderate Highest cost Requires environment 
development Highest degree of modification of development 
effort realism the FSW to stub (stubs to cut 
(cost/schedule) Limited fault out board support subsystem out of 
Provides high injection options dependencies system arch. may 
degree of realism Depends on HW May require less require extensive 
(hard to refute and SW effort to build development) 
findings) availability May be available * Not TEST as Fly 
Dependent on TEST as Fly from ¢ Provides quick 
build releases of development assessment of 
software under program unit level 
test Not ground concerns. 

Most flexible test system dependent ¢ Algorithm 
options verification 
TEST as Fly 


Test Environment Acquisition Methods 


ett i Project Test Environment 


/2 | Develop Test Environment (Hybrid or Completely Independent) 


Test Environment Remote Access 


HWIL Environment 


Test Environment Development Methodology 


¢ Users must specify FSW and Database versions 


¢ Users must specify spacecraft hardware configuration 


Typical User-Supplied 
Inputs 


Test Environment 
Outputs 
Spacecraft Taksvaaeleatcvale 
Models Models —+Faral 
GR) om \V/Kole(=1 owe lalemeyian 
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WW YoYo (=) (=Xe. Instruction Set 


Spacecraft ; 
“ Spacecraft Simulator 
Dynamics Simics/Qemu 
aElachW ela 


Gyros, Reaction ppc750, ppc401, LEON3, etc 


Wheels, Solar Panels, cPCl, Spacewire, 
Etc. 1553, Ka-band, S- 


band, FPGAs, etc 


Memory 
Analysis 


Faulttnjection 
What-lIf 
Y see 


Scenarios eo 
Independent 
Testin 
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Value 


Raw Values 
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Test Execution and Analysis 


¢ Execute, Collect Results, and Analyze Results 


Adjust Test Planning as necessary for next 
iteration 


ae & tes) pre culOD are aa BAT 


ild and release test environment in sequential iterations 
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Increased system and software understanding 


Increased IV&V Program Capability 
Measurable IV&V assurance 


Run-time experience with most critical flight software 
behaviors 


Validation of fault handling software 


Verification of software behavior and identification of 
software issues 
— Often process forces flight software down non-happy paths 
¢ High Severity Software Issue Found During Model Development 


* Execution of “non-flight” code in the flight binary 


¢ Flight computer model forced flight software down non-happy paths of the 
board support package (BSP) 
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